Designing a Modern Application Data Stack (for Hoang Anh) by Adam Morton Brad Culberson Kevin McGinley

Author:Adam Morton, Brad Culberson, Kevin McGinley
Language: eng
Format: epub, pdf
Publisher: O'Reilly Media, Inc.
Published: 2023-10-27T00:00:00+00:00

Workload Isolation

Workload isolation is a critical component for cloud data platforms supporting data applications, especially because the challenges of resource contention expand in multi-tenant environments.

In on-premise data warehouses, multiple workloads often compete for the same resources. For example, extract, transform, and load (ETL) processes are generally resource intensive, requiring large amounts of CPU, memory to process and store data, and I/O and network bandwidth when writing and reading data to the storage layer.

If data ingestion and user queries share the same hardware, a large data ingestion event could saturate the compute resources and cause poor performance for user queries. The workaround has been to run the ETL activities outside the business day so they didn’t impact the users and applications running queries. Today we have data arriving far more frequently, making this traditional and rigid approach impractical.

Your cloud data platform should enable you to establish clusters of designated resources that can be aligned with tasks and workloads of a similar profile. It should also enforce transactional consistency so that data is never in an invalid state when multiple workloads are running on the same data concurrently.

In general, the apps that support business-critical activities get more compute resources set aside for them. Although this assures enhanced performance, you must be mindful of the trade-off, which is a higher cost.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.